Skip to content

feat(clickhouse): add the ClickHouse destination package - #46

Merged
gjtorikian merged 12 commits into
mainfrom
feat/sync-clickhouse
Nov 6, 2026
Merged

gjtorikian merged 12 commits into
mainfrom
feat/sync-clickhouse

Conversation

@gjtorikian

Copy link
Copy Markdown
Collaborator

Summary

  • Adds @workos/sync-clickhouse, a ClickHouse destination with the same CLI, container image and Destination semantics as the Postgres and MySQL packages, aimed at what ClickHouse is for: keeping the authentication.* sign-in log and the event archive past the 90-day retention, with the identity objects mirrored beside them for the joins.
  • Mirror streams are ReplacingMergeTree(_updated_at) tables with _deleted_at tombstones and generated _live views (FINAL, tombstones hidden); append streams are plain MergeTree. Projected columns are DEFAULT expressions over _raw (JSONExtract(... 'Nullable(T)'), parseDateTime64BestEffortOrNull), with data-skipping indexes where the catalog asks for one.
  • The lease, cursors, schema fingerprints and column records live in KeeperMap tables, the one place ClickHouse offers a compare-and-swap: strict insert-if-absent and conditional ALTER UPDATE as CAS, verified on a local 26.10 and on ClickHouse Cloud 26.6. That is the source of the package's one extra requirement, a Keeper plus keeper_map_path_prefix, which the README states at the technology level alongside the single-node config CI uses.
  • Reconcile diffs inside the database through per-run staging tables; the quarantine store is a ReplacingMergeTree cleared with lightweight DELETE; redact uses lightweight DELETE across the mirror, the sign-in log and the archive.
  • Two ClickHouse-specific repairs the tests surfaced: a column added to an existing table is MATERIALIZEd so old parts stop re-evaluating the expression on every read, and init --repair follows MODIFY COLUMN with an UPDATE ... WHERE 1, since MODIFY COLUMN alone does not recompute stored values.
  • Ports --exclude-streams and the oversized-stream reporting from feat: Abandon oversized reconcile streams, add excludeStreams #45 into this CLI.

Testing

164 tests run against a clickhouse/clickhouse-server:26.10 container started with the config in test/clickhouse/ (embedded Keeper, keeper_map_path_prefix); the pure tests run without one and the rest skip. CI gains a ClickHouse job, the Docker workflow gains the clickhouse matrix entry, and release-please gains the package. The image builds and streams runs from it.

Mirror streams are ReplacingMergeTree tables with tombstones and _live
views; append streams are plain MergeTree. The lease, cursors and schema
bookkeeping live in KeeperMap tables, the one compare-and-swap ClickHouse
offers, so a Keeper and keeper_map_path_prefix are required. Projected
columns are DEFAULT expressions over _raw, materialized when added to an
existing table and recomputed on repair.

Wires the package into CI (a ClickHouse container with an embedded
Keeper), the Docker matrix, release-please and the READMEs.
@greptile-apps

greptile-apps Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; the latest version changes introduce no blocking issue.

Summary

Adds @workos/sync-clickhouse with a CLI, database writes, schema setup, reconciliation, redaction, tests, and release wiring.

  • ClickHouse stores WorkOS identity rows and event history for analysis.
  • ClickHouse syncs keep their cursor and writer lease in KeeperMap.
  • Operators can compare the ClickHouse mirror with WorkOS and repair differences.
  • Bad events can be set aside and replayed after their cause is fixed.
  • A ClickHouse sync can run from its own CLI and report its health.
  • The ClickHouse package joins the repository's build and release workflows.

Reviews (9) · Last reviewed commit: "chore(clickhouse): start sync-clickhouse..." · Reviewed by Greptile

Comment thread release-please-config.json Resolved
…ckpoint races

Every mirror write is read-decide-write, and ClickHouse has no row lock
to serialize it, so two writers on one key (the tail beside a backfill,
or a poll outliving its lease) judged against the state before the
other's insert landed: a merge could drop fields, a tombstone could lose
a version tie, an append-only event could land twice. Each batch now
holds a per-stream lock in a `_locks` KeeperMap table: strict-mode
INSERT to take it, DELETE to release, conditional UPDATE to take over
one a crashed process left behind once it expires. A lock wait that
runs out is a transient error, so the engine retries rather than
quarantines.

`migrate` left a view alone whenever it had every column; a foreign
definition with every column now goes through `applyLiveView` and is
refused, or replaced under `--repair`. A checkpoint that Keeper refused
over a version race (the holder's own renewal is enough) re-reads and
retries instead of reporting a moved cursor and replaying the poll.

Also: release-please calls the publish workflow for this package, and
the test server config loses the `--` XML forbids inside a comment, which
kept CI's ClickHouse from starting.
…ones at deletion time

A batch that stalled past the lock's TTL could still insert rows judged
against state another writer has since changed. It now checks it still
holds the lock right before writing and fails with LOCK_LOST, a transient
error the engine retries from a fresh read.

An object delete's tombstone takes the deletion time as its version
instead of the row's last update time, so it outranks any older snapshot
on the version itself rather than on which insert landed last. The
deletion came after anything a listing could still show of the row, so
nothing legitimate is outranked.
# Conflicts:
#	.github/workflows/release-please.yml
… out-of-range offsets

The same three fixes Postgres (#49) and MySQL (#50) got. npm installs the
bin as a symlink named after the command, which the old argv check did
not recognise, so `npx workos-sync-clickhouse` imported the CLI and did
nothing. Duplicate keys in a batch are now judged by the instant each
timestamp names rather than by its text, so `01:00:00+01:00` no longer
outranks `00:30:00Z`. And the timestamp projection range-checks the
clock and the offset ahead of the best-effort parser, which read
`24:00` as the next midnight and `+00:99` as a zone where the other
destinations project NULL.
…ate append rows

`init --repair` redefines a column, and its `_live` view keeps the type it
declared when it was created: a query through the view then returns the
old type or fails to convert. The views of the streams a repair touched
are now re-created after it, and the check that leaves an up-to-date
view alone compares column types as well as names.

Append-only tables are a ReplacingMergeTree keyed by the event id, so a
second copy of an event collapses at the next merge, and under FINAL
before then. The write path's existence check under the stream lock
still keeps copies out in the first place; the table is the backstop
for a poll that outlived its lease.
# Conflicts:
#	.github/workflows/release-please.yml
#	.release-please-manifest.json
#	README.md
#	packages/sync-engine/README.md
…ng the schema

Two processes applying the same catalog at once both record the same
column clauses and the same fingerprint. A KeeperMap insert decides
create-or-set per key before its transaction, so with strict mode off
the second writer can still be refused with "Node exists" when the key
appeared in between, and `init` failed on it. Both records now read what
landed: the same value counts as recorded, a different column clause is
overwritten in place.
…rded columns

`redact --user` erased rows by `user_id` only. An invitation names a user
as its inviter and as the user who accepted it, so those rows, and the
archived and quarantined events carrying those payloads, kept the
subject's id and email after the command reported success. Every column
and payload field that names a user is matched now.

A column with no `_columns` record had its expected clause recorded
without a look at the table, so after a schema applied by hand from
`init --print-sql` a changed path was recorded as installed, with the
column still reading the old path. The catalog's clause now goes through
the server's own formatter, by way of a throwaway Memory table, and is
compared with the column's stored expression; a match is recorded and
anything else is drift that `--repair` rewrites.
… probe table

Checking an unrecorded column built a throwaway table, which `init` had
no business needing a `DROP TABLE` grant for. The server's formatter is
a function too: `formatQuerySingleLine` spells a clause exactly as
`system.columns.default_expression` does, so the comparison is one
query with no DDL. The README's grants gain `DROP TABLE` for the staging
tables `reconcile` has always built and dropped, and the version floor
moves to 23.11, where the function arrived.
The same seed as the other packages (#51): the first version is
published by hand so the npm trusted publisher can be configured, and
release-please reads 0.0.0 as never released and would propose 1.0.0,
where from 0.0.1 the next release computes to 0.1.0.
@gjtorikian
gjtorikian merged commit a8ea9e9 into main Nov 6, 2026
23 checks passed
@gjtorikian
gjtorikian deleted the feat/sync-clickhouse branch November 6, 2026 14:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant